Performance Comparison of Apache Spark and Tez for Entity Resolution

Entity Resolution is among the hottest topics in the field of Big data. It finds duplicates in datasets, which actually belong to same entity in the real world. Algorithms that perform Entity Resolution are computation intensive and consume a lot of time especially for large datasets. A lot of research has been conducted for improving Entity Resolution solutions. A number of algorithms are developed, in attempt to reduce the time required to execute Entity Resolution algorithms on a given dataset. Efficiency of Entity Resolution algorithms has significantly improved but is still not adequate for large datasets in the Big data field. We are contributing to enhance its performance in terms of time, not by improving the algorithm but finding the most suitable platform on which it runs. This would, in turn, increase its efficiency and indirectly elevate the accuracy of Entity Resolution by empowering it to run more computation intensive algorithm. We have shortlisted Apache Spark(RDD, DataFrame and Dataset) and Apache Tez (Hive) as the set of platforms. In this research work we have chosen the Blocking technique for implementing Entity Resolution in the four above mentioned different applications. We have performed a number of experiments with different configurations to find the most efficient platform by analyzing, comparing and evaluating the results in great detail.